Our human brains are particularly good at solving problems when we are able to make use of our physical and propriocentric intuition, yet current computer interfaces make little use of these abilities. The hypothesis underlying the present invention is that interaction mediated by active computer-controlled objects will improve understanding and collaboration in many types of simulations for which screen-based interaction is not optimal. Current methods to affect such a capability are either expensive or limited in important ways.
The present invention is a practical, flexible and low cost planar manipulator display that can simultaneously move dozens of physical objects upon a surface under computer control and sense movement of those objects by users, as shown in FIGS. 1 and 3. Specifically, (i) Each of many physical objects can be moved quickly, accurately and independently upon a surface, (ii) the positions of the objects can be accurately sensed (iii) the mechanism scales gracefully to surfaces of large area, and (iv) the cost per movable object does not exceed several dollars, thereby enabling widespread application in laboratories, classrooms, and eventually the home.
Such a device could be used in conjunction with a projection display, which projects information upon the surface, and applications in which users' directives are recognized via vision-based gesture recognition and voice recognition. Work between two or more co-located collaborators can be aided by such an information tool, by making use of shared proprioception (body-space awareness).
Applications could include military scenario simulation, studies of the flow and interaction of people in social or emergency evacuation situations, automotive traffic testing and evaluation, interactive algorithms for arrangement of furniture or architecture, and interactive science education, one example being a physically responsive kit of optical components that can form themselves into functional arrangements under user control.
There is ample precedent to show that a rethinking of the physical interface to the computer can lead to a profound change in the use of computers in society. This is logical: Having evolved as physical creatures, our reasoning skills are tightly coupled to our perceptual skills. For example, as computer output has shifted from low fidelity text displays to high resolution full color displays, there has been a corresponding shift not only in the way we interact with computers, but in our very uses of computers. A striking example of this has been the recent great increase in computer use by the general populace, and the rapid and widespread adoption of the World Wide Web that occurred as soon as hyperlinked images were implemented in browsers.
The benefits of passive (non-actuated) physical objects in a user interface have been demonstrated by many researchers, including [Robert J. K. Jacob, Hiroshi Ishii, Gian Pangaro, and James Patten, A Tangible Interface for Organizing Information Using a Grid CHI 2002 Conference on Human Factors in Computing Systems Minneapolis, Minn. 20-25 Apr. 2002; James Patten, Hiroshi Ishii and Gian Pangaro: Sensetable: A Wireless Object Tracking Platform for Tangible User Interfaces CHI 2001 Conference on Human Factors in Computing Systems Seattle, Wash., USA Mar. 31-Apr. 5, 2001; Hiroshi Ishii, Brygg Ullmer: Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. CHI 1997: 234-241; G. Fitzmaurice, H. Ishii, and W. Buxton, “Bricks: Laying the Foundations for Graspable User Interfaces, Proceedings of CHI'95 (1995), pp. 442-449, all of which are incorporated by reference herein.
Several other research groups have made enabling technologies for planar manipulation of physical objects. Each of these systems has a particular limitation or deficiency which precludes implementation at reasonable cost for simultaneous planar manipulation of many objects.
The only approach that has been demonstrated to work on simultaneous planar transport of multiple objects is the Universal Planar Manipulator by Dan Reznik at Berkeley [D. Reznik, “The Universal Planar Manipulator”, Ph.D. Thesis, UCBerkeley, EECS, October 2000; D. Reznik and J. Canny, “Universal Part Manipulation in the Plane with a Single Horizontally-Vibrating Plate”, 3rd International Workshop on Algorithmic Foundations of Robotics (WAFR), Houston, Tex., March, 1998, both of which are incorporated by reference herein.] This system creates small vibratory movement of the surface, exploiting the non-linearity of friction to “shake” objects along a table surface. By time-slicing the vibration patterns, each object can be addressed individually and made to follow an independent trajectory. The major advantages of this approach are that it works with any object, and that it is relatively inexpensive.
The deficiency of this approach is that maximum speed of object movement decreases linearly with the number of objects. Since the frequencies used require approximately 10 milliseconds to address one object, only a small number of objects can be simultaneously moved at interactively useful speeds. Also, the presence of an unavoidable continual vibration of the entire table precludes the use of this system where such vibration would be considered objectionable. The system is also inherently limited to be operable only on horizontal surfaces.
The Actuated Workbench at MIT is a table consisting of a packed array of electromagnets. By varying the relative strengths of these magnets, ferromagnetic objects can be manipulated by being handed along between neighboring magnets [Gian Pangaro, Dan Maynes-Aminzade, Hiroshi Ishii The Actuated Workbench: Computer-Controlled Actuation in Tabletop Tangible Interfaces UIST 2002 Symposium on User Interface Software and Technology Paris, France, Oct. 27-30, 2002, incorporated by reference herein]. The advantage of the actuated workbench is its lack of moving parts, and the fact that every location on the surface always contains an actuator.
Its deficiencies include a relatively weak effective force (only movement of carefully chosen lightweight objects has been demonstrated) and large weight of the packed electromagnets, which increases linearly with surface area.
The Virtual Vehicle is a tabletop packed with a checkerboard array of computer-controlled motor-actuated protruding rollers that alternate between two orthogonal directions (rollers at even squares are perpendicular to those at odd squares). Each protruding roller is independently drivable; objects are translated or rotated by varying the rotation of subsets of rollers [J. Luntz, W. Messner, and H. Choset Virtual Vehicle: Parcel Manipulation and Dynamics with a Distributed Actuator Array Proceedings of the SPIE International Symposium on Intelligent Systems and Advanced Manufacturing, Sensors and Controls for Advanced Manufacturing, Vol. SPIE 3201, 1997, incorporated by reference herein].
This mechanism shares with the Universal Planar Manipulator the property that unprepared objects may be transported—objects need not be ferromagnetic.
Disadvantages are that the mechanical complexity and cost are relatively high, increasing linearly with unit area. Only a small demonstration unit has been made; it is not clear that it would be practical to scale this device up to cover a large surface.
The Courier Robot project at CMU [A. Quaid and A. Rizzi Robust and Efficient Motion Planning for a Planar Robot Using Hybrid Control IEEE International Conference on Robotics and Automation 2000, Vol. 4, Apr. 2000, pp. 4021-4026; R. Hollis and A. Quaid An Architecture for Agile Assembly American Society of Precision Engineering 10th Annual Mtg, Oct. 1995, both of which are incorporated by reference herein] consists of a two directional planar (Sawyer) motor in which the stator is an entire tabletop and the rotor is a self-contained vehicle running a sophisticated closed-loop control that rides atop a very thin air gap. This approach allows extremely rapid and finely controlled movement. However, courier robots are really designed for the speed and precision required for precision assembly in miniature table-top factories. The high cost per vehicle, and per unit area of tabletop, as well as the power cabling needed for each vehicle, preclude their use in the user interface context which is the focus of this proposal.